Looking into reading II:a follow-up study on test-takers' cognitive processes while completing Aptis B1 reading tasks

This study investigated 25 ESL test-takers’ cognitive processing while completing a set of Opinion Matching tasks designed and piloted for use on the Aptis reading test and targeting the CEFR-B1level. Insights were gained through the recording of participants’ eye traces during task completion, immediately followed by a stimulated recall after each task in which participants described, in their first language, how they had completed it.The study follows up on the findings of Brunfaut & McCray (2015), who investigated test-takers’ task processing on the full Aptis reading test, and found support for the construct validity of the test. However, a somewhat weaker alignment between the intended and actual reading processes used by test-takers was found for the B1 banked gap-fill tasks of the Aptis reading test. The aim of this follow-up study, therefore, was to explore the cognitive processing on an alternative, newly designed set of Opinion Matching tasks to be able to evaluate the extent to which the new tasks elicited the specified processes for this level of the Aptis reading test (see O’Sullivan & Dunlea, 2015).Test-takers were found to use a wide range of cognitive processes while they were completing the B1 Opinion Matching tasks, covering both lower- and higher-level processes as defined in Khalifa & Weir’s (2009) model of reading. Most often, when they successfully solved the item, the test-takers had adopted a careful and global reading approach (and sometimes they also did some expeditious reading), and they had combined lexical processing and/or meaning-making within sentences, across sentences or at textual and intertextual levels with inferencing. This global comprehension approach and the extensive engagement with higher-level processing associated with the B1 Opinion Matching items not only differs substantially from the local and lower-level processing patterns associated with the original B1 banked gap-fill items, but also suggests a suitable match with the intended processes specified by the test developers for the Aptis B1 reading tasks

Investigating the construct measured by banked gap-fill items:evidence from eye-tracking

This study investigates test-takers’ processing while completing banked gap-fill tasks, designed to test reading proficiency, in order to test theoretically based expectations about the variation in cognitive processes of test-takers across levels of performance. Twenty-eight test-takers’ eye traces on 24 banked gap-fill items (on six tasks) were analysed according to seven on-line eye-tracking measures representing overall, text and task processing. Variation in processing was related to test-takers’ level of performance on the tasks overall. In particular, as hypothesised, lower-scoring students exerted more cognitive effort on local reading and lower-level cognitive processing in contrast to test-takers who attained higher scores. The findings of different cognitive processes associated with variation in scores illuminate the construct measured by banked gap-fill items, and therefore have implications for test design and the validity of score interpretations

Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront

In this invited Viewpoint on the occasion of the 40th anniversary of the journal Language Testing, I argue that at the core of future challenges and opportunities for the field – both in scholarly and operational respects – remain basic questions and principles in language testing and assessment. Despite the high levels of sophistication of issues looked into, and methodological and operational solutions found, outstanding concerns still amount to: what are we testing, how are we testing, and why are we testing? Guided by these questions, I call for more thorough and adequate language use domain definitions (and a suitable broadening of research and testing methodologies to determine these), more comprehensive operationalisations of these domain definitions (especially in the context of technology in language testing), and deeper considerations of test purposes/uses and of their connections with domain definitions. To achieve this, I maintain that the field needs to continue investing in the topics of validation, ethics, and language assessment literacy, and engaging with broader fields of enquiry such as (applied) linguistics. I also encourage a more synthetic look at the existing knowledge base in order to build on this, and further diversification of voices in language testing and assessment research and practice

Exploring the role of phraseological knowledge in foreign language reading

Foreign language (FL) knowledge has been shown to contribute significantly to FL reading performance. Studies have contrasted the contribution of FL vocabulary and syntactic knowledge, following a dichotomous view of these components, producing mixed results. Despite the increasingly recognized formulaic nature of language, the contribution made by phraseological knowledge to reading ability has not been investigated systematically. This study examines the impact of a broader construct definition of linguistic knowledge – which includes a phraseological component – in explaining variance in reading performances. Test scores of 418 learners of English as a foreign language (EFL) were modeled in a structural equation model, showing that a phraseological knowledge measure outperformed traditional syntactic and vocabulary measures in predicting reading comprehension variance. Additional insights into the role of phraseological knowledge were gained through verbal protocol analysis of 15 EFL learners answering reading comprehension items that targeted the understanding of phrasal expressions within written context. The findings hint at an underestimated, but critical, role of phraseological knowledge in FL reading, and are relevant to both the assessment and the teaching of EFL ability

Diagnostic assessment of reading and listening in a second or foreign language:elaborating on diagnostic principles

Alderson, Brunfaut and Harding (2014) recently investigated how diagnosis is practised across a range of professions in order to develop a tentative framework for a theory of diagnosis in second or foreign language (SFL) assessment. In articulating this framework, a set of five broad principles were proposed, encompassing the entire enterprise of diagnostic assessment. However, there remain questions about how best to implement these principles in practice, particularly in identifying learners’ strengths and weaknesses in the less well-documented areas of SFL reading and listening. In this paper, we elaborate on the set of principles by first outlining the stages of a diagnostic process built on these principles, and then discussing the implications of this process for the diagnostic assessment of reading and listening. In doing so, we will not only elaborate on the theory of diagnosis with respect to its application in the assessment of these skills, but also discuss the ways in which each construct might be defined and operationalized for diagnostic purposes

Towards a theory of diagnosis in second and foreign language assessment:insights from professional practice across diverse fields

Diagnostic language assessment has received increased research interest in recent years, with particular attention on methods through which diagnostic information can be gleaned from standardized proficiency tests. However, diagnostic procedures in the broader sense have been inadequately theorized to date, with the result that there is still little agreement on precisely what diagnosis in second and foreign language learning actually entails. In order to address this problem, this article investigated how diagnosis is theorized and carried out in a diverse range of professions with a view to finding commonalities that can be applied to the context of language assessment. Ten semi-structured interviews were conducted with professionals from the fields of car mechanics, IT systems support, medicine, psychology and education. Data were then coded, yielding five macro-categories that fit the entire data set: (i) definitions of diagnosis, (ii) means of diagnosis, (iii) key players, (iv) diagnostic procedures, (v) treatment/follow-up. Based on findings within these categories, a set of five tentative principles of diagnostic language assessment is drawn-up, as well as a list of implications for future research

Bridging assessment and learning:a view from second and foreign language assessment

This paper considers issues around the relationship between assessment and learning, as put forward by Baird, Andrich, Hopfenbeck and Stobart (2017), from the perspective of the field of second and foreign language assessment. In our response, we describe shared observations on the nature of research and practice in general educational assessment and in language assessment (including with respect to linking assessment with theories of learning, managing impact, and enhancing assessment literacy). At the same time, we also identify areas where language assessment seems to diverge from current research and practice in general educational assessment (for example in the areas of assessment purposes, construct definitions, and validation theory and practice). As a consequence, we believe that close monitoring of advances in both fields is likely to be mutually beneficial

Bridging assessment and learning:a view from second and foreign language assessment

This paper considers issues around the relationship between assessment and learning, as put forward by Baird, Andrich, Hopfenbeck and Stobart (2017), from the perspective of the field of second and foreign language assessment. In our response, we describe shared observations on the nature of research and practice in general educational assessment and in language assessment (including with respect to linking assessment with theories of learning, managing impact, and enhancing assessment literacy). At the same time, we also identify areas where language assessment seems to diverge from current research and practice in general educational assessment (for example in the areas of assessment purposes, construct definitions, and validation theory and practice). As a consequence, we believe that close monitoring of advances in both fields is likely to be mutually beneficial